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(54) Qos nnonltoring system and method for a high-speed diffserv-capable network element 



(57) A QoS nfionltoring system and method for a Diff- 
Sen^-capable networl< element operable in a trusted do- 
main network such as an ISP network. The network el- 
ement Is organized as a plurality of tenninating line 
cards interconnected via a switch fabric capable of sup- 
porting virtual ingress/egress pipes (VIEPs). Buffer 
queues on the Ingress and egress sides of the network 
element, which are established for supporting traffic 
flows on individual VIEPS, are monitored for determin- 
ing QoS parametric information such as throughput, 
loss, delay, jitter and available bandwidth. A policing 



structure is operably coupled with a buffer acceptance 
and flow control module for monitoring traffic behavior 
on the ingress side. Another buffer acceptance/flow 
control module and aggregate-level monitoring module 
are disposed on the egress side of the network element 
that cooperates with a scheduler which shapes outgoing 
traffic. The monitoring for the PIPE traffic reflects the 
confonnance of the service provider to their customers, 
whereas the monitoring for the HOSE traffic reflects the 
level of over- or under-provisioning for a given COS. 
Feedback flow control is provided between the ingress 
and egress sides for throttling buffer acceptance. 
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Description 

BACKGROUND OF THE INVENTION 
Technical Field of the Invention 

[0001] The present invention relates generally to 

Quality of Service (QoS) provisioning in communica- 
tions networks. More particularly, without limitation, the 
present invention relates to a QoS monitoring system 
and method for a high-speed DiffServ-capable network 
element (e.g., a router) disposed in an autonomous sys- 
tem. 

Description of Related Art 

[0002] Driven by the myriad advances in networidng 
technology that are taking place at an unprecedented 
rate, Internet access solutions and Internet-based ap- 
plications such as e-commerce have become the main- 
stay of today's New Economy. Internet service providers 
(ISPs) and Internet access providers (lAPs), which pro- 
vide access to the Intemet and range in size from small, 
local operators to national entitles offering connectivity 
and IP-based services nationwide or internationally in 
some instances, are now as ubiquitous as local hard- 
ware stores and compete vigorously for subscribers by 
offering a variety of pricing plans and variable bandwidth 
services. Further, an increasing number of prominent 
national ISPs have begun to offer proprietary services 
and content in addition to simple Internet access. 
[0003] Despite the Internet's rapid growth over the 
last ten years or so, several important considerations 
remain. For example, because the Intemet is a connec- 
tionless and stateless network, current Internet-based 
sen/ice implementations can only provide "best-effort" 
services. That is, whereas the network will try its best to 
fonward user traffic, it cannot provide any guarantees re- 
garding packet loss rate, bandwidth, delay, etc. Thus, 
packets may be dropped Indiscriminately In the event of 
congestion, path failures, and the like. While this kind of 
service works fine for some traditional applications (e. 
g., file transfer protocol or FTP, email, etc.), it is intoler- 
able for the newly emerged real time, multimedia appli- 
cations such as Internet Telephony, video-conferencing, 
video-on-demand. Interactive TV (ITV), online music, 
etc. 

[0004] Accordingly, it is commonly understood in the 
communications industry that the cornerstone of future 
IP network growth will be IP QoS, which provides for a 
set of service requirements to be met by the IP network 
while transporting a flow (typically defined as a packet 
stream from a source to a destination (unicast or multi- 
cast)). In other words, QoS is defined as a measurable 
level of service delivered to network users, which can 
be characterized by a set of metrics (e.g., packet loss 
probability, delay, jitter or delay variation, available 
bandwidth, et cetera). Such QoS can be provisioned by 



network service providers in ternis of a service agree- 
ment (e.g., a Service Level Agreement or SLA) between 
subscribers and providers. For example, a subscriber's 
requirement can be that for some traffic flows generated 
5 by the subscriber, the network should guarantee a path 
with at least certain bandwidth level. 
[0005] It should be apparent that by employing differ- 
ent levels of IP QoS, service providers can achieve 
greater profitability through premium services offered to 
high-margin business customers, more efficient use of 
network resources, and higher-priced service levels. In 
addition, they can be more competitive through en- 
hanced service differentiation, better-than-best-effort 
service, and customized solutions. 
[0006] To make a contractual agreement that custom- 
ers can trust, a service provider needs a network with 
QoS capabilities and a policy management system to 
configure, control, and maintain perfonnance levels. Dif- 
ferentiated Services (DiffServ) is an IP QoS architecture 
defined by the Intemet Engineering Task Force (IETF) 
that has particular reference to the service provider and 
earner networks. DiffServ concentrates on aggregating 
flows and per hop behavior applied to a network-wide 
set of traffic classes, thereby minimizing the amount of 
signaling required. Effectively, DiffServ provides a light- 
weight signaling mechanism between service provider's 
domain borders and network nodes, carrying infomia- 
tion about each packet's service requirements. 
[0007] Whereas the DiffServ framework provides 
broad architectural guidelines with respect to the provi- 
sioning of IP QoS in a trusted domain, management of 
traffic flows within an individual DiffServ-capable node 
is contemplated to be application- and implementation- 
specific. As a result, there exists a need for solutions 
that reliably and accurately monitor the traffic behavior 
within a node and help determine QoS relevant para- 
metric information in order to ensure appropriate levels 
of service within the DiffServ frameworic. 
[0008] Cun'ent techniques for monitoring intra-nodal 
traffic behavior for DiffServ purposes are beset with var- 
ious shortcomings and deficiencies, however. For ex- 
ample, the existing QoS monitoring schemes typically 
involve processes with a high granularity of measure- 
ments. Thus, the aggregate level traffic behavior (e.g., 
per port, per class, etc.) is not adequately captured. In 
addition, where the traffic Is segregated Into different 
queues according to some classification, dynamic be- 
havior of such queues Is not monitored against service- 
constraint-based thresholds that may be required for 
SLA assurance, compliance and analysis. As a conse- 
quence, the current solutions cannot provide a reliable 
measurement of average occupancy of the DiffServ- 
provisioned queues. Furthermore, parameters that 
quantify resource-specific behavlorsuch as average un- 
der- and over-utilization of the resources (e.g., band- 
width, buffer depth, etc.) are not adequately profiled as 
well. 
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SUMMARY OF THE INVENTION 

[0009] Accordingly, the present Invention provides a 
QoS monitoring system and method for a DlffServ-ca- 
pable network element operable in a trusted domain net- 
work (such as an ISP/IAP network) that advantageously 
overcomes these and other shortcomings of the state- 
of-the solutions. Preferably, the trusted domain network 
Is operable as an autonomous system wherein QoS par- 
ametric Infomiatlon may be monitored on multiple ag- 
gregate levels for SLA analysis, compliance and en- 
forcement. 

[0010] In one aspect, the present invention is directed 

to a networi< element (e.g., an edge router, core router, 
or transit router, collectively, a routing element) that Is 
organized as a plurality of temriinating tine cards orTLKs 
Interconnected via a switch fabric capable of supporting 
virtual ingress/egress pipes (VIEPs) between transmit- 
ter cards (Ingress cards) and receiver cards (egress 
cards). Each TLK card is operable to support one or 
more incoming or outgoing communication links with re- 
spect to the networi< element, depending on Its config- 
uration. At least a portion of the TLK cards are operable 
as the networic element's Ingress side. SImllariy, a por- 
tion of the TLK cards are operable as the egress side of 
the network element. Buffer queues on the ingress and 
egress sides of the network element, which are estab- 
lished for supporting traffic flows on individual VIEPs, 
are monitored for detemilning QoS parametric Informa- 
tion such as throughput, loss, delay, jitter and available 
bandwidth. A policing stmcture Is associated with the 
ingress cards for monitoring and measuring Incoming 
traffic on the incoming communications links against an 
expected traffic profile or behavior pattern associated 
with the Incoming traffic. A buffer acceptance and flow 
control module is associated with each of the Ingress 
and egress cards that operates to manage the traffic 
flows associated with the VIEPs through the switch fab- 
ric. Preferably, the traffic flows are operable to be effec- 
tuated with resource reservations allocated in the switch 
fabric depending on type of service (e.g., real time vs. 
non-real time), Class of Service, SLA-based traffic en- 
gineering (TE) policies/priorities, et cetera. A traffic 
shaping and scheduling module Is operable with an ag- 
gregate-level monitoring module disposed on the 
egress cards for scheduling and shaping outgoing traffic 
on the outgoing communications links to the networic el- 
ement's neighboring nodes in the network. Feedback 
flow control is provided between the ingress and egress 
sides for throttling buffer acceptance and packet dis- 
carding based on buffer congestion thresholds estab- 
lished on the egress side. 

[0011] In another aspect, the present invention Is di- 
rected to a method for processing QoS parametric in- 
formation In a networic element operable In an IP net- 
woric, wherein the network element Includes at least one 
terminating line card operable as an Ingress card sup- 
porting an incoming communications link, at least one 



temnlnatlng line card operable as an egress card sup- 
porting an outgoing communications link and a switch 
fabric disposed between the Ingress and egress cards 
for supporting a plurality of VIEPs therebetween. Upon 

5 receiving incoming infonnation packets on the incoming 
link of the network element, a detennlnation is made in 
an Ingress portion a networic processor system dis- 
posed on the ingress card whether the incoming infor- 
mation packets pertains to an IP-based service. Re- 

10 sponslve to the detemnlning step, the Incoming informa- 
tion packets are fonwarded to an egress portion of the 
network processor system via the switch fabric. The 
packets are monitored for confomiance with respect to 
the reserved VI EP resources to the destination TLK (i. 

IS e., egress card). The processed Information packets are 
transmitted to the egress card via a select VIEP for rout- 
ing the processed infonnation on a target outgoing link 
to a neighbor in the network. The egress portion prefer- 
ably includes an embedded processor operable to per- 

20 form a plurality of IP-based QoS (IPQoS) monitoring op- 
erations and for processing the incoming infonmation in- 
to processed information. 

BRIEF DESCRIPTION OF THE DRAWINGS 

25 

[0012] A more complete understanding of the present 
Invention may be had by reference to the following De- 
tailed Description when taken in conjunction with the ac- 
companying drawings wherein: 

30 [0013] FIG. 1 depicts an exemplary autonomous sys- 
tem operating as a trusted domain for coupling with a 
plurality of networks, wherein network elements incor- 
porating the teachings of the present invention are ad- 
vantageously employed; 

35 [0014] FIG. 2 depicts a functional block diagram of an 
exemplary network element provided In accordance 
with the teachings of the present invention for operating 
in an trusted domain; 

[0015] FIG. 3 depicts a functional block diagram of a 
40 network processor subsystem used in a tenninating line 
card (TLK) of the exemplary network element of the 
present Invention; 

[0016] FIG. 4 depicts a functional block diagram of 
packet flow in the exemplary network element; 
45 [0017] FIG. 5 is a message flow diagram for effectu- 
ating IP QoS monitoring in the exemplary networic ele- 
ment In accordance with the teachings of the present 
invention; 

[0018] FIG. 6 depicts a functional block diagram of a 
50 QoS monitoring system for use in the exemplary net- 
work element In accordance with the teachings of the 
present Invention; 

[001 9] FIG. 7 depicts exemplary color monitors used 
as a component In a DiffServ traffic conditioner provided 
55 in the exemplary network element in accordance with 
the teachings of the present Invention; 
[0020] FIGS. 8A - 8C depict various packet discarding 
mechanisms that may be utilized as a component in flow 
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control for controlling traffic flows within the exemplary 
network element; 

[0021] FIG. 9 depicts a policing mechanism for 
throughput at an ingress TLK of the exemplary network 

element; 

[0022] FIG. 10 depicts a functional block diagram of 
a flow control system for use in the exemplary network 
element in accordance with the teachings of the present 
invention; and 

[0023] FIG. 1 1 depicts exemplary metric profiles mon- 
itored at an egress TLK of the DiffSen^-capable network 
element of the present invention. 

DETAILED DESCRIPTION OF THE DRAWINGS 

[0024] In the drawings, like or similar elements are 
designated with Identical reference numerals through- 
out the several views thereof, and the various elements 
depicted are not necessarily drawn to scale. Refemng 
now to FIG. 1 , depicted therein is an exemplary auton- 
omous system (AS) network 102 operating as a trusted 
domain for coupling with a plurality of networks, wherein 
a plurality of network elements incorporating the teach- 
ings of the present invention are advantageously em- 
ployed. It should be appreciated by those skilled in the 
art that the AS network 1 02 is preferably provided as a 
routing domain which has a common administrative au- 
thority and consistent internal routing policy. In an ex- 
emplary embodiment, the AS network 102 may employ 
multiple intradomain routing protocols internally (e.g., 
Open Shortest Path First (OSPF), Routing Infomiation 
Protocol (RIP), etc.) and interface to other ASs via a 
common interdomain routing protocol (e.g., Border 
Gateway Protocol or BGP). 

[0025] In addition, the AS network 102 may be pro- 
vided in an exemplary functional embodiment as an ISP/ 
lAP network operated by a network service/access op- 
erator for providing various IP-based services, Including 
access, In accordance with any established or hereto- 
fore unknown Differentiated Services (DlffSen^) scheme 
that supports IP-based QoS (IPQoS). As an ISP net- 
work, accordingly, the AS network 102 is operable to 
serve its subscribers via a plurality of networks, e.g., vir- 
tual private networks (VPNs) 1 1 0, peer networks 1 08 (e. 
g., other ISP networks), enterprise or corporate net- 
works 109 (e.g., intranets), and circuit-switched net- 
works or packet-switched networks for dial traffic, e.g., 
network 112. Further, a public IP network such as the 
Intemet 113 is also coupled to the AS network 102 for 
facilitating Internet-based services involving data, 
voice, multimedia, and video. 
[0026] A plurality of DIffServ-capable network ele- 
ments or nodes (e.g., edge routers 104A-104E and tran- 
sit routers 106A-106D) fomri the trusted domain of the 
AS network 1 02, which is capable of instituting a range 
of SLAs with one or more of its subscribers including 
dial-up, corporate, wholesale, or peer network custom- 
ers. These SLAs may be simple standard service con- 



tracts for mass consumers or customized and multidi- 
mensional service agreements for business and corpo- 
rate customers. An SLA, which defines end-to-end serv- 
ice specifications, may comprise any of the following 

5 components in the context of the present invention: (i) 
sen/ice availability; (ii) service levels offered; (iii) service 
guarantees; (Iv) responsibilities; (v) service auditing; 
and (vi) pricing an-angements. 
[0027] A plurality of QoS metrics or measurements 

10 are preferably used for quantifying the service require- 
ments of a particular SLA. Well known QoS metrics in- 
clude bandwidth (BW), throughput, delay, jitter (i.e., de- 
lay variation), cost, loss probability or packet loss rate, 
et cetera. These QoS metrics may be categorized into 

IS three types: additive, multiplicative, and concave. Let m 
(n1 ,n2) be a metric for Imk(n1 ,n2). For any path P = (n1 , 
n2, . . . ni, nj), where n1, n2, . ., nj represent networi< 
nodes, metric m is additive if m(P) = m(n1 ,n2)+m(n2,n3) 
+ . . .+m(ni,nj). Examples are delay, jitter, cost, and hop- 

20 count. For instance, the delay of a path is the sum of 
delay of every hop. Metric m is multiplicative if m(P) = 
m(n1,n2)*m(n2,n3)* . . , *m(ni,nj). An example of a mul- 
tiplicative metric is reliability, in which case 0 < m(P) < 
1. Metric m Is concave if m(P) = min{m(n1,n2), m 

25 (n2,n3), . . .,m(ni,nj)}. Available bandwidth is an exam- 
ple of a concave metric, where the bandwidth of a path 
is determined by the link hop with the minimum available 
bandwidth. 

[0028] As will be described in greater detail hereinbe- 

30 low, QoS metric monitoring may be effectuated at one 
or more DiffServ-capable network elements of the AS 
network 1 02 for any of the following Classes of Service 
(COS) as may be applicable: Constant Bit Rate (CBR); 
Real Time Variable Bit Rate (VBR-RT); Non-Real Time 

35 Variable Bit Rate (VBR-N RT); Available Bit Rate (ABR); 
and Unspecified Bit Rate (UBR). 
[0029] FIG. 2 depicts a functional block diagram of an 
exemplary network element 200 provided In accordance 
with the teachings of the present invention for operating 

40 In a trusted domain such as the AS network 102 de- 
scribed hereinabove. The network element 200 is pref- 
erably comprised of a plurality of temnination line cards 
(TLKs), e.g., TLK 202A and TLK 202B. and a plurality 
of real time server (RTS) boards 210, wherein the TLK 

45 cards and RTS boards are Interconnected through a 
switching fabric 204. In the presently preferred exem- 
plary embodiment of the present invention, the switch- 
ing fabric 204 is provided as a Multi-Path Self Routing 
(MPSR) switch that Is capable of supporting a plurality 

50 of virtual Ingress/egress pipes (VIEPs) used for trans- 
porting traffic flows through the network element. 
[0030] In addition to the internal communication path- 
ways established through the MPSR switch fabric 204 
(which is preferably used for all IP load and control traf- 

55 fic), the TLK cards and RTS boards are operable to com- 
municate via an overiay network 220 used for adminis- 
trative functions such as software downloading, initiali- 
zation, and management and maintenance configura- 
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tion. A management server (MS) 21 8 is accordingly pro- 
vided as part of the network element 200 for coordinat- 
ing and hosting these administrative functions.. 
[0031] The functionality of TLK cards Includes tennl- 
nation of external communication linlcs, IP/MPLS for- 
warding, tennination of link-related protocols (e.g., RPR, 
Label Distribution Protocol or LDP, Resource Reserva- 
tion Protocol or RSVP, etc.) and switch interface func- 
tions such as Segmentation and Reassembly (SAR), re- 
sequencing, etc. The RTS boards complement forward- 
ing capabilities of the TLK cards by providing the capa- 
bility to process routing protocol messages. Routing 
protocols such as BGP, OSPF, etc., are typically proc- 
essed on a route server (RS) 213 on the RTS boards. 
Consequently, forwarding tables are calculated and dis- 
tributed through the M PSR switch fabric 204 by the route 
server to all fonvarding engines on the TLK cards of the 
network element. In addition, the RTS boards are used 
for relaying external management messages from any 
external interface to MS 218. Further, an Interface (e.g., 
Gigabit Ethernet (GENET) interface) (not explicitly 
shown In FIG. 2) to an external "charging server" may 
be included in the RTS boards for effectuating pricing 
policies of an SLA. 

[0032] The functionality of the TLKs and RTSs is pri- 
marily carried out by one or more network processor 
(NP) modules (e.g., reference numeral 214) in conjunc- 
tion with an on-board controller (OBC) processor 212. 
Each NP module is preferably comprised of an ingress 
portion 21 5A and an egress portion 21 5B. As will be 
seen In greater detail hereinbelow, an embedded proc- 
essor (EP) 216 provided in the egress portion 21 5B is 
primarily responsible for the processing of incoming 
packets having IP service options (Including IPQoS 
monitoring requirements). Moreover, EP 21 6 is also op- 
erable to process signaling protocol packets (origination 
or destination) for both L2 and L3 layers (e.g., PPP and 
GENET at L2 and RSVP and LDP at L3). 
[0033] In addition to interfacing with the overlay com- 
munication network 220, OBC processor 212 Is respon- 
sible for MPSR Interface control. The switch generic In- 
terface functionality of the TLK/RTS card is comprised 
of a traffic manager (TM), which may be provided as a 
separate TM module 216 or as an embedded function- 
ality of OBC processor 212, and a switch component. 
TM functionality Is primarily responsible for effectuating 
the interface between the MPSR switch and the TLK/ 
RTS card at all levels: physical level, logical-protocol 
level, and BW management level. 
[0034] QoS-aware or QoS monitoring software appli- 
cations running on EP 216 and OBC 212 are operable 
to inter-communicate via a TCP/IP protocol stack of res- 
ident operating systems (OS). For example, Infomriation 
regarding BW allocation for VIEPs in the switch is pref- 
erably communicated from an RSVP application (which 
Is an EP process) to OBC in order to properly configure 
a TLK's TM. Further, the software environment of both 
processors preferably includes appropriate drivers (e. 



g., Peripheral Component Interconnect (PCI) drivers, 
etc.) for additional functionality. 
[0035] Referring now to FIG. 3, depicted therein is a 
functional block diagram of the network processor sub- 

5 system 214 In additional detail. A networking function 
(NF) 232 is responsible for packet processing function- 
alities such as, e.g., forwarding, filtering, scheduling, 
etc. in addition to processing information packets with 
IP options, EP 216 is also operable to pertomn control 

10 functions such as I P control protocol message process- 
ing, exception processing, table management, and ex- 
ecuting link-related signaling protocols. Several func- 
tional Interfaces are associated with the NP subsystem 
214 for facilitating its networking and QoS-aware func- 

15 tionalltles. An external link interface 236 is provided for 
supporting incoming or outgoing links (e.g., links 206 
and 208 depicted in FIG. 1) with respect to the networtc 
element. When configured as a receiver for packet in- 
fonnation emanating from transmitting neighbors, the 

20 TLK having the NP module 214 is operable as an In- 
gress card disposed on the ingress side of the network 
element. In similar fashion, a TLK having the NP module 
214 may be configured as an egress card disposed on 
the egress side of the networi< element when packet in- 

25 fomiation is transmitted via the external link interface 
236 to the neighboring receiver elements. The external 
link interface 236 is therefore operable to receiveArans- 
mit packets towards the PHY layer devices that can be 
configured to support Layer 2 protocols, e.g.. Fast Eth- 

30 ernet (FENET), GENET, etc., with appropriate media ac- 
cess control (MAC). 

[0036] A switch interface 238 is provided for transmit- 
ting to or receiving from the switch fabric various intra- 
node traffic flows that are managed for QoS assurance. 

35 In some exemplary embodiments, a redundant switch 
interface may also be provided for increased reliability. 
A control memory array 234 Is interfaced with the NP 
module 21 4 for storing control information, e.g., fonward- 
ing infonnatlon base or bases (FIB), QoS monitoring 

40 counters, etc. Preferably, the control memory array 234 
may be comprised of both SRAM and DRAM. 
[0037] Continuing to refer to FIG. 3, a data buffer 
memory 240 Is interfaced to the NP module 21 4 for stor- 
ing infonnation packets at the egress before they are 

45 transmitted on the extemal link or towards the switch 
fabric. Preferably, the data buffer memory 240 is Imple- 
mented with double data rate (DDR) DRAM modules. A 
PCI interface 242 Is provided for connecting an external 
host processor 244 such as, e.g., OBC processor 212 

50 depicted in FIG. 2. Those skilled in the art should rec- 
ognize that this Interface is primarily used for system 
initialization and interaction of the NP module 214 with 
board and system management functions. 
[0038] FIG. 4 depicts a functional block diagram of IP 

55 packet flow in the exemplary network element of the 
present invention. On the ingress side, the header of re- 
ceived frames Is first parsed by a hardware (HW) clas- 
sifier 302. This process classifies the packet depending 
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on code entry point (e.g., PPP vs. GENET frame) and 

protocol number in L2 header into different protocol 
types (e.g., IP over PPP, MPLS over PPP, PPP control 
packet flow/et cetera). In addition, the HW classifier 302 
detects a plurality of predesignated fields in the IP head- 
er (e.g. , color of DiffServ class (described in greater de- 
tail below), TCP/UDP/TCP_SYN flag, IP service flags, 
etc.). The results of this classification process are 
passed to appropriate software (S W) modu les at the dis- 
patch time as Code Instruction Address or Addresses 
(CIA) and a Flow Control Block (FCB) page. 
[0039] After dispatching the frames to the SW mod- 
ules, they are analyzed In three stages: L2, L3 and L4 
processing (reference numerals 304, 306 and 308, re- 
spectively). L2 processing is based on the MAC desti- 
nation address (DA) and involves parsing the protocol 
number in l_2 header (IP, Address Resolution Protocol 
(ARP), PPP control messages, etc.). If the frame is a 
PPP control message, it is redirected to the EP of the 
NP module (shown in FIG. 2). Where IP packets are in- 
volved, they are passed to L3 processing 306, together 
with an indication as to where IP header starts. 
[0040] L3 processing 306 involves performing Long- 
est Prefix Match (LPM) search on IP DA. In a presently 
prefen-ed exemplary embodiment of the present inven- 
tion, various checks are also perfomned on the IP head- 
er: IPv4, checksum, etc. In addition, several operations 
relating to DiffServ (policing, buffer acceptance and con- 
trol, BW classification, et cetera, which will be set forth 
In greater detail hereinbelow) are perfomned at this 
stage as well. Three possible outcomes can be gener- 
ated due to L3 processing: (i) f onward IP packets to L4 
processing; (II) redirect the frames to another module 
(e.g., IP with sen/ice options is redirected to the EP of 
the NP module); or (ill) drop the frame. 
[0041] L4 processing 308 involves Multi-Field (MF) 
classification, wherein a searching key is constructed 
based on multiple fields of IP and TCP headers and a 
software managed tree (SMT) search Is then performed. 
Once the processing is completed and the contents of 
the FCB page fields are filled in, the frame Is enqueued 
In the ingress scheduler for scheduling towards the 
switch fabric (reference numeral 310). At the transmis- 
sion time (after the frame has been scheduled), appro- 
priate HW constructs a Frame Header (FH) and Cell 
Header (CH) based on the FCB fields, and the data is 
read from the ingress buffer and sent to the switch fabric 
in the fomn of cells having a predesignated fomiat (ref- 
erence numeral 312). 

[0042] At the egress side, a frame reassembly proc- 
ess 314 is effectuated first. The reassembled frame is 
dispatched for HW classification (reference numeral 
316) where the HW classifier parses the FH and sets 
appropriate code entry points depending on the frame 
format. Subsequently, a Next Hop (NH) processing 
block 318 perfonns a search on the NH IP address in 
the ARP table. The result of the search returns L2 en- 
capsulation (reference numeral 320) to be used. Other 



checks such as fragmentation detection (depending on 
Maximum Transmission Unit (MTU) size, frame altera- 
tion commands (e.g., new IP checksum), etc., may also 
be performed at this time. Thereafter, L2 encapsulation 

s con^esponding to a target port Is added to the IP packet 
and the frame is enqueued in a flow queue of the egress 
scheduler (reference numeral 322). At the time the 
frame is eligible for transmission, it is read from the 
egress data buffer and transmitted on an extemal link 

10 interface. 

[0043] To support DiffSen^ (DS) capabilities, a Type 
of Service (ToS) field available in IPv4 implementation 
is marked on the ingress side In order to effectuate var- 
ious levels of sen/ice priorities. The eight-bit ToS field is 

15 treated as a DS field where priorities such as DE (De- 
fault - indicating a best-effort CoS); EF (Expedited For- 
warding; AF (Assured Forwarding), etc., are mapped. 
To support the various DS levels, cell traffic in the VIEPs 
is also mapped with appropriate levels. Priority 00 and 

20 priority 01 cells are mapped Into high priority queues (re- 
al time (RT) traffic). As a consequence, during port con- 
figuration, adequate BW reservation needs to be set up 
for the appropriate VI EP In order to prevent higher loss 
of RT traffic in that VIEP On the other hand, low priority 

25 cells (i.e.. Priority 10) are mapped to non-real time 
(NRT) queues. As pointed out eariier, ensuring premium 
sen/lces requires a QoS metrics monitoring system and 
method relevant to the traffic flow within the network el- 
ement, which will be set forth in greater detail hereinbe- 

30 low. 

[0044] FIG. 5 is a message flow diagram for effectu- 
ating IP QoS monitoring in the exemplary networi< ele- 
ment In accordance with the teachings of the present 
invention. TLK 202A and TLK 202B are exemplified as 

35 the ingress and egress sides of the network element. 
Upon receiving infonmatlon packets on an incoming link 
402, detennlnation is made (reference numeral 404) in 
the ingress NP portion 21 5A of the ingress TLK 202A 
whether IP service options are involved. Also, the in- 

40 coming traffic is policed for in-prof lie orout-of-profile de- 
terminations. If the packets involve IP options, they are 
redirected (reference numeral 406) to the egress NP 
portion 21 5B via the switch fabric, where the EP appro- 
priately processes the packets for egress scheduling 

45 (including detemrilnlng outgoing Interface andtarget port 
information) (reference numeral 408), Also, various 
QoS-aware and QoS-specific monitoring applications 
relating to buffer acceptance and VI EP-level flow control 
are perfomned. The processed infonnation Is then 

50 passed back (reference numeral 41 0) to the ingress por- 
tion of the ingress TLK 202A, whereupon it is transmitted 
(reference numeral 412) to the egress NP portion 21 5B 
of the egress TLK 202B through the switch fabrfc. Ap- 
propriate buffer control takes place on the egress side 

55 also (reference numeral 414), which may preferably In- 
volve sending feedback control signals to the ingress 
TLK for throttling its buffer control mechanism. In addi- 
tion to partial egress processing, PIPE/HOSE-level 
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monitoring (described hereinbelow in additional detail) 
also takes place at this juncture. Thereafter, the outgo- 
ing traffic Is shaped and scheduled for transmission (ref- 
erence numeral 416) on an outgoing link. 
[0045] FIG. 6 depicts a functional block diagram of a 
QoS monitoring system 500 for use in the exemplary 
network element in accordance with the teachings of the 
present invention. For purposes of ensuring DiffServ ca- 
pability and con-esponding SLA-based service con- 
straints, various resource-based parametric monitors 
are advantageously employed as part of the QoS mon- 
itoring system of the present invention. For example, 
parametric infonmation such as average occupancy of 
buffer queues, average over- and under-utllization of 
BW, etc. is deployed in orderto manage appropriate ag- 
gregate-level QoS metrics. In a presently preferred ex- 
emplary embodiment of the present invention, these 
metrics Include throughput, loss, delay, jitter, and avail- 
able BW. 

[0046] Throughput is defined as the average amount 
of data (in bytes and packets) transferred to a destina- 
tion point per CoS. This measure is utilized to set up, 
manage, and identify different thresholds on the bulk of 
traffic flow per CoS. It should be appreciated that by em- 
ploying per flow, per threshold levels, this measure be- 
comes critically useful for effectuating a proactive action 
on the traffic behavior. Loss may be defined as the ratio 
of the amount of data dropped (in bytes and packets) to 
the amount of data transferred to a destination point per 
CoS. Accordingly, this metric measures the behavior of 
the buffer queues allocated to a particular traffic flow 
against their current reservation (i.e., queue utilization). 
Further, this metric also identifies to some extent the dy- 
namic behavior of the queues and assists in perfomiing 
reactive actions on the traffic behavior. 
[0047] Delay is measured as the queuing delay In the 
system for different types of behavior. In addition to in- 
stantaneous values, the average behavior of this pa- 
rameter is also important which depends on the CoS 
type. The average buffer queue depth is computed as 
the average of the instantaneous depths of the queue 
taken over a period of time. Jitter is defined as the var- 
iation or variance of the queuing delay. Average queue 
occupancies may be measured in relation to jitter mon- 
itoring in order to arrive at better measurements for the 
resource behavior. Available BW is the unreserved BW, 
which is monitored per link for traffic engineering pur- 
poses. 

[0048] In orderto monitor these QoS parametrics, the 
present invention provides structures and techniques 
for measuring the traffic characteristics on the ingress 
side as well as the egress side of the networi< element. 
A policing structure 504 is provided in the ingress TLK 
202A which accepts a plurality of flows 502 (having dif- 
ferent types) in order to measure the incoming traffic 
against the expected behavior. Traffic entering the Dlff- 
Serw domain, wherein the network elements are provid- 
ed in accordance with the teachings of the present in- 



vention, needs to be classified for appropriate treatment 
inside the domain. It must either be pre-marked by the 
customer or mari<ed at the edge router level on the serv- 
ice provider's side of the network demarcation point. 

5 [0049] Classification of customer traffic by the service 
provider's edge router can be based on multiple criteria, 
ranging from the interworking of various priority 
schemes to application level analysis of traffic within the 
IP packets as set forth hereinabove. Traffic policing may 

10 be implemented using a classifier (for classifying the in- 
coming traffic), a token bucket or similar mechanism (for 
monitoring entry traffic levels at each class), and mark- 
ers (for identifying or downgrading non-compliant traf- 
fic). FIG. 9 depicts a policing mechanism for throughput 

15 at an ingress TLK of the exemplary networic element. As 
shown In FIG. 9, throughput monitoring is effectuated 
by tracking the in-profile and out-of-profile measure- 
ments over a time period. Throughput measurements 
are plotted against time as a profile 802 and a threshold 

20 804 is defined for separating the in-profile portion 806B 
from the out-of-profile portion 806A. It should be appre- 
ciated that such aggregate-level throughput measure- 
ments suffice because of the incoming traffic flows at 
wire-speed. 

25 [0050] The policing function is preferably effectuated 
by the NP module at both HW and code levels. A plu- 
rality of policing filters are provided as part of the policing 
structure 504, wherein one or more policing actions per 
packet are available. Further, policing may also be per- 

30 formed in accordance with a three-color maricer (TCM) 
(described hereinbelow in reference to FIG. 7). Addition- 
ally, the loss parameter is measured as a projection of 
the traffic profile from the previous nodes that generate 
the incoming traffic towards the network element. 

35 [0051] Continuing to refer to FIG. 6, a buffer accept- 
ance and flow control module (hereinafter, a flow con- 
troller) is provided on both ingress and egress TLK 
cards. For example, flow controller 506 is provided as 
part of the functionality of the NP module (which may be 

^0 referred to as the UP ramp NP module) of the ingress 
TLK 202A. Similarly, flow controller 510 is provided as 
part of the NP module (the DOWN or DN ramp NP mod- 
ule) of the egress TLK 202B. A QoS-aware traffic shap- 
er/scheduler 508 is operable in association with the flow 
controller 51 0 on the egress TLK 202B for appropriately 
loading the outgoing links in accordance with QoS- 
based policies and constraints. 
[0052] To monitor the characteristics of the traffic on 
the ingress and egress sides, various counters are im- 

50 plemented in association with the QoS-aware modules 
described hereinabove. Counters 506 are provided for 
the ingress TLK that measure (i) packets and bytes 
transferred per egress TLK per queue type and (ii) pack- 
ets and bytes dropped per egress TLK per queue type. 

55 In similar fashion, counters 512 are provided for the 
egress TLK for measuring (i) packets and bytes trans- 
mitted in the egress TLK per neighbor per queue (ac- 
counts for the throughput); (ii) packets and bytes 
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dropped in the egress TLK per neighbor per queue (ac- 
counts for random drops and tail drops); (iii) average 
queue depth (to account for the delay and jitter param- 
eters); and (iv) number of times a threshold is crossed 
on a packet discard probability profile. 
[0053] FIG. 7 depicts exemplary color monitors used 
as a component in a DiffServ traffic conditioner for the 
policing functionality provided In the exemplary network 
element in accordance with the teachings of the present 
invention. Reference numeral 600A refers to a single- 
rate (sr) TCM and reference numeral 600B refers to a 
two-rate (tr) TCM. The srTCM 600A meters an IP packet 
stream and marks (re-marks) its packets either green 
(G), yellow (Y), or red (R). iVIarking is based on a Com- 
mitted Information Rate (CIR) and two associated burst 
sizes, a Committed Burst Size (CBS) and an Excess 
Burst Size (EBS). A packet is marked green If it doesn't 
exceed CBS, yellow if exceeds the CBS, but not the 
EBS, and red othenwise. Thus, the srTCIVI is Implement- 
ed as a dual leaky bucket metering device (bucket 602 
and bucket 604) in conjunction with a marker/re-marker 
for marking the incoming packets depending on their 
conformance to specified traffic profiles. 
[0054] The trTCM 600B meters an IP packet stream 
and marks/re-marks its packets based on two rates, 
Peak Information Rate (PIR) and CIR. A packet is 
marked red if it exceeds the PIR value. Otherwise, it is 
marked either yellow or green depending on whether it 
exceeds or doesn't exceed the CIR value. 
[0055] The srTCM is useful for Ingress policing of a 
service where only the length, not the peak rate, of the 
burst detemiines service eligibility. On the other hand, 
trTCM is useful, for example, for ingress policing of a 
service where a peak rate needs to be enforced sepa- 
rately from a committed rate. Either TCM is operable in 
one of two modes. In color-blind mode, the meter as- 
sumes that the packet stream is uncolored. In the color- 
aware mode, the meter assumes that some preceding 
entity has pre-colored the incoming packet stream so 
that each packet is either green, yellow, or red. 
[0056] Additional details regarding the TCMs may be 
found in the Internet Engineering Task Force's RFC 
2697 and RFC 2698 which are incorporated by refer- 
ence herein. 

[0057] The TCMs 600A and 600B can be used to 
mark a packet stream in a service, where decreasing 
levels of assurances (either absolute or relative) are giv- 
en to packets depending on their color. For example, a 
service may discard all red packets, because they ex- 
ceeded both CIR and CBS, fonvard yellow packets as 
best effort, and fonvard green packets with a low drop 
probability (e.g., AF traffic). 

[0056] The functionality of the policing structure 504 
(depicted in FIG. 6) is at least partly based on a suitable 
TCM algorithm wherein the TCM-resultant color is inter- 
preted locally by the software in order to effectuate dif- 
ferent policing actions, for example: (i) no action; (ii) dis- 
card immediately; (iii) rewrite the DS field (i.e., re-mark); 



or (Iv) rewrite the traffic type field (used for defining the 
drop probability of the packet locally inside the network 
element). 

[0059] FIGS. 8A - 8C depict various packet discarding 

5 mechanisms that may be utilized as a congestion avoid- 
ance component in flow control for controlling the traffic 
flows within the exemplary network element of the 
present invention. Reference numeral 702 in FIG. 8A 
refers to a conventional random early discard (RED) 

10 profile that is based on TCP's congestion control mech- 
anism. By randomly discarding packets prior to high 
congestion (preferably within certain boundaries of the 
average queue level, Qmin 706 and Qmax 708), the 
RED mechanism instructs a TCP source to decrease its 

15 transmission rate. In an exemplary embodiment, the 
RED mechanism is preferably implemented using an 
exponentially weighted average of a queue level, a dis- 
card probability curve (e.g., probability profile 702), and 
a buffer deep enough to absorb short-temn bursts. 

20 [0060] Reference numeral 704 refers to a dampened 
RED profile which critically dampens the buffer dynam- 
ical system, thereby avoiding oscillatory behavior of the 
queue level. Another modified RED mechanism in- 
volves applying different RED probabilities on a per-flow 

25 threshold basis. For example, depending on whether 
the packets belong to a flow that is over- or under- 
shared, different probabilities may be applied. This var- 
iant is particularly useful (in conjunction with a QoS 
aware scheduler) to protect in-profile flows from drops 

30 due to RED (i.e., discard probability (DP) = 0 If flow 
queue level < TH, threshold). 
[0061 ] FIGS. 8B and 8C depict packet discard mech- 
anisms wherein packet-type awareness is employed. In 
a Weighted RED (WRED) mechanism, different RED 

35 drop probabilities apply, depending on the type of the 
received packets (e.g., QoS class, AF drop precedence, 
responsiveness, et cetera). Reference numerals 710, 
712 and 714 in FIG. 8B refer to the DP profiles for red, 
yellow and green packets, respectively. Reference nu- 

40 merals 71 6, 71 8 and 720 in FIG. 8C refer to the DP pro- 
files for packets in TCP_SYN, User Datagram Protocol 
(UDP), and TCP flows, respectively. 
[0062] In addition to the TCMs and discard mecha- 
nisms set forth above, BW provisioning mechanisms are 

45 provided as part of the overall QoS monitoring scheme 
of the present invention. A Reserved/Unreserved (R/U) 
scheduler is included which can be programmed to pro- 
vide a mix of reserved and unreserved BW in the net- 
work element. Each flow gets a granted BW (GBW) plus 

50 a share of the excess BW in proportion to an adminis- 
trative weight (AW), wherein excess BW is computed as 
any BW instantaneously unused (which equals [non-re- 
served BW] + [reserved, but currently unused GBW] + 
[currently unused AW]). 

55 [0063] Further, MPSR internal flow control is utilized 
for distributing through-switch BW among contenders. 
A Connection Admission Control (CAC) module and In- 
ternal Dynamic Rate-based Flow Control (IDRFC) mod- 



15 



EP1227 624 A2 



16 



ule are provided as part of the Internal flow control that 
operates at the granularity of VIEPs. The CAC module 
distributes reserved BW pipes per VIEP based on peri- 
odic negotiation among all contending VIEPs. The IDR- 
FC module is responsible for allocating non-reserved 
BW, which is distributed fairly among contending VIEPs 
in accordance with a Need For Bandwidth (NFB) reso- 
lution mechanism (which is preferably based on the UP 
ramp's per-VlEP buffer occupancy/arrival rate statis- 
tics). 

[0064] As part of the policing functionality of the QoS 
monitoring system, the incoming traffic flows are classi- 
fied, differentiated, and then broadly categorized into RT 
and RT queues. Depending on classes, traffic differen- 
tiators, etc., resource provisioning (e.g., buffers, BW 
and the like) for the RT and NRT traffic is done in ac- 
cordance with QoS-based policies and constraints. For 
example, in setting up an RT flow which is preferably 
modeled in the "PIPE" model (where entry point and exit 
point of customer traffic is known), the following actions 
are taken: (i) GBW Is reserved for a particular target port 
on the egress TLK card; (ii) CAC module associated with 
the egress TLK reserves this GBW for the VIEP associ- 
ated with the RT flow; (lii) CAC module associated with 
the ingress TLK reserves appropriate GBW for the cor- 
responding VIEP; and (Iv) the policer parameters per 
target are updated in the ingress side. The NRT traffic 
is provisioned both In the PIPE model as well as the 
"HOSE" model (entry point is known but the exit point 
may be Indeterminate). For setting up the NRT flows, 
the egress side NP module Is first configured with GBW, 
AW, CBS, and PIR parameters. Optionally, per-class 
thresholds may then be configured therefor. Also, both 
ingress side and egress side CAC modules may be con- 
figured to resen^e appropriate GBW for the VIEP asso- 
ciated with the NRT flow. 

[0065] The QoS monitoring module of the present in- 
vention Is operable to measure the behavior of the traffic 
due to the various reservations in the switch fabric per 
VIEP between the Ingress and egress forwarding en- 
gines. Thus, the functionality of the buffer acceptance/ 
flow controller module on the ingress side Involves man- 
aging the queue behavior dynamics in the context of the 
egress scheduling, incoming traffic flows, and BW con- 
sumption In the switch fabric. The monitoring for the 
PIPE traffic reflects the confomriance of the service pro- 
vider to their customers, whereas the monitoring for the 
HOSE traffic reflects the level of over- or under-provi- 
sioning for a given COS. 

[0066] Referring now to FIG. 1 0, depicted therein Is a 
functional block diagram of a flow controller system 900 
for use in the exemplary network element in accordance 
with the teachings of the present invention. Ingress TLK 
card 202A and egress TLK card 202B are exemplified 
once again as the ingress and egress sides of the net- 
work element. Each side Is provided with CAC (which is 
configurable via appropriate signaling messages from 
the neighboring nodes) and IDRFC modules for BW res- 



ervation, VIEP arbitration, and Internal flow control. In 
the exemplary embodiment depicted in FIG, 10, IDRFC 
906A and CAC 908A are associated with the ingress 
side TLK and, In similar fashion, IDRFC 906B and CAC 

5 908B are associated with the egress side TLK. 

[0067] As described In detail hereinabove, policer 504 
of the ingress side TLK Is operable in conjunction with 
a packet discard structure 902A In order to condition the 
incoming traffic 502 for classification, differentiation, 

10 and categorization. A local congestion Indicator 904A Is 
provided to be operable in conjunction with the policer 
and packet discard mechanisms. Multiple flows targeted 
to the egress side TLKs are set up as a plurality of 
queues 91 OA, both RT (reference numeral 91 2A) and 

15 NRT (reference numeral 91 4A), wherein data buffers 
are allocated to each queue based on a flow control al- 
gorithm. It should be appreciated that the plurality of 
queues on the ingress TLK are indexed in temis of per 
egress card and per flow type (e.g., RT vs NRT). 

20 [0068] Similarly, a plurality of queues 91 OB are set up 
on the egress side TLK for the outgoing traffic 509 em- 
anating from the network element. Preferably, at least 
eight queues per neighbor are set up in a presently pre- 
ferred exemplary embodiment of the present invention. 

25 These queues are indexed In accordance to per target 
port, per CoS, and per flow type. Thus, reference nu- 
merals 91 2B and 914 refer to RT and NRT queues for 
the egress TLK 202B. A local congestion Indicator 904B 
is provided to be operable In conjunction with the egress 

30 side discard structure 902B, which In turn Is associated 
with traffic shaper and scheduler 508. 
[0069] Two factors are predominant In the measure- 
ment of the traffic behavior on the Ingress side due to 
flow control, namely, (i) the flow control between the 

35 TLKs and (11) the queuing for the RT and NRT queues 
between the TLKs. A data buffer to a particular egress 
TLK Is accepted if the egress TLK is not congested. This 
infomiation is received as a feedback flow control signal 
91 3 from the egress congestion Indicator 904B to a tar- 

40 get threshold comparator 911 disposed in the ingress 
TLK 202A. It should be appreciated that where the 
number of buffers In these queues Is sufficiently low, the 
delay and jitter parameters may be neglected in some 
implementations. However, under these low buffer num- 

45 bers, the measurement of throughput and loss param- 
eters becomes more significant. As the QoS module is 
operable to allocate resources through the switch for RT 
and NRT queues, throughput measurements can be ad- 
vantageously used to detemriine whether the allocation 

50 for these two types of traffic is sufficient, both in quali- 
tative and quantitative terms. Where there is an Interac- 
tion between the RTand NRT sources, such interactions 
may be due to the condition that switch resource allo- 
cation cannot detemrilned because the traffic cannot be 

55 characterized (e.g., incapable of identifying high priority 
routing traffic) or the traffic characteristics cannot be de- 
temilned a priori (traffic modeled on the HOSE model 
and where there is no signaling Involved). 
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[0070] The loss parameter provides the amount of 

packet loss in the switch due to under allocation of re- 
sources per queue, per egress ILK. Average queue 
depth measurements represent the delay characteris- 
tics between the TLKs. As alluded to in the foregoing, 
where the number of buffers is very limited, the delay 
and jitter measurements may be Ignored. 
[0071] The QoS monitoring module of the present in- 
vention is also operable to measure the outgoing traffic 
behavior for better resource management. The shaper 
and scheduler 508 is thus provided to be QoS-aware 
and the discard structure 902B operable with the egress 
TLK flow controller Is preferably provided to be a highly 
enhanced RED type mechanism with class, color and 
connection type awareness. FIG, 11 depicts exemplary 
metric profiles that can be monitored at an egress TLK 
of the DiffServ-capable network element of the present 
invention. Throughput 1 002, queue depth 1 004 and loss 
1006 are profiled for the queue behavior on the egress 
side. Egress side buffer acceptance constitutes a mech- 
anism to accept or drop packets and If packets are to 
be dropped, how they have to be dropped (e.g., RED, 
WRED, or modified RED, etc.). Once a packet is accept- 
ed Into the queue, it will be scheduled according to the 
different scheduler configuration parameters. Thus, a 
queue buildup is quite possible because of the various 
scheduling parameters. 

[0072] As the monitoring of flow queue utilization is 
critical for DiffServ aggregation, the present invention's 
QoS monitoring module is also operable to monitor 
whether DiffServ flow queues are over- or under-uti- 
lized. In the presently preferred exemplary embodiment 
of the present Invention, this monitoring Is done by com- 
paring total reserved BW for a DiffServ flow queue with 
the actual usage of allocated resources. The actual re- 
source usage by a flow queue Is determined from the 
total number of bytes fon^^arded over a period of time. 
For UP ramp monitoring, each flow queue per neighbor 
is monitored for different UP ramp thresholds. For ex- 
ample, these thresholds may be yellow (85%) and red 
(90%). Similarly, DN ramp monitoring is accomplished 
by setting up different DN ramp thresholds (e.g., blue 
(65%) and white (50%) levels). In addition, both over- 
utilization and under-utilization traps may be set up for 
these multiple UP and DN ramp thresholds. 
[0073] Based on the foregoing, those skilled in the art 
should appreciate that the present Invention provides an 
innovative IPQoS monitoring solution that advanta- 
geously quantifies over- and under-utilized resources 
and delay measurements for a DiffServ-capable routing 
element of an autonomous network. By monitoring ag- 
gregated QoS parameters such as average occupancy 
of queues, average utilization levels, etc. and comparing 
these aggregated parameters against established 
thresholds, a better resource provisioning model is ob- 
tained for effectuating DiffServ capability in a more effi- 
cient manner. Further, QoS-relevant parametric infor- 
mation obtained from the monitoring system of the 



present Invention Is particularly useful for building and 
analyzing end-to-end SLAs. 

[0074] It is believed that the operation and construc- 
tion of the present Invention will be apparent from the 

5 foregoing Detailed Description. While the system and 
method shown and described have been characterized 
as being preferred. It should be readily understood that 
various changes, modifications and enhancements 
could be made therein without departing from the scope 

10 of the present Invention as set forth In the following 
claims. 



Claims 

15 

1. A network element operable in an Internet Protocol 
(IP) network, comprising: 

at least one terminating line card operable as 
20 an ingress card supporting an incoming com- 

munications link; 

at least one terminating line card operable as 
an egress card supporting an outgoing commu- 
nrcatlons link; 

25 a switch fabric disposed between said ingress 

and egress cards for supporting a plurality of 
virtual ingress/egress pipes (VIEPs) therebe- 
tween, wherein said ingress and egress cards 
are provided with a plurality of buffer queues to 

30 support traffic flows associated with said 

VIEPs; 

a policing structure disposed on said ingress 
card for monitoring incoming traffic on said in- 
coming communications link with respect to an 
3s expected traffic profile associated with said in- 

coming traffic; 

a flow controller disposed on each of said in- 
gress and egress cards, said flow controllers 
operating to manage said traffic flows assocl- 

^0 ated with said VIEPs through said switch fabric, 

wherein said traffic flows are operable to be ef- 
fectuated with resource reservations allocated 
In said switch fabric; and 
a scheduler disposed on said egress card for 

"^5 scheduling and shaping outgoing traffic on said 

outgoing communications link. 

2. The network element operable in an IP network as 
set forth in claim 1 , wherein said policing structure, 

50 said flow controllers, and said scheduler operate in 
concert with a plurality of counters associated with 
said ingress and egress cards to provide Quality of 
Service (QoS) parametric infomriation necessary to 
effectuate a Service Level Agreement between a 

55 networic service provider operating said networic el- 
ement and a subscriber for services available in 
said IP network. 



19 

3. The network element operable In an IP network as 

set forth in clainri 2, wherein each of said flow con- 
trollers is associated with at least one local conges- 
tion indicator, and further wherein each of said flow 
controllers operate in conjunction with a packet dis- 
carding mechanism for throttling flow on a particular 
VIEP 

4. The network element operable in an IP network as 
set forth in claim 3, wherein said packet discarding 
mechanism associated with said flow controller on 
said ingress card is operable to be controlled at 
least in part by a feedback signal received from said 
at least one local congestion indicator disposed on 
said egress card. 

5. The network element operable in an IP network as 
set forth in claim 3, wherein said plurality of buffer 
queues are categorized as one of real time (RT) and 
non-real time (NRT) queue types. 

6. The network element operable in an IP networi< as 
set forth in claim 5, wherein said plurality of counters 
associated with said Ingress card Include a counter 
for monitoring the number of packets transferred 
per egress card per queue type. 

7. The network element operable in an IP networic as 
setforth In claim 5, wherein said plurality of counters 
associated with said ingress card Include a counter 
for monitoring the number of bytes transferred per 
egress card per queue type. 

8. The network element operable in an IP network as 
setforth in claim 5, wherein said plurality of counters 
associated with said Ingress card include a counter 
for monitoring the number of packets dropped per 
egress card per queue type. 

9. The network element operable in an IP networi< as 
setforth in claim 5, wherein said plurality of counters 
associated with said ingress card include a counter 
for monitoring the number of bytes dropped per 
egress card per queue type. 

10. The network element operable in an IP network as 
set forth In claim 5, wherein said QoS parametric 
infonnation comprises traffic flow throughput infor- 
mation per Class of Service (CoS) per destination. 

11. The network element operable In an IP networi< as 
set forth in claim 5, wherein said QoS parametric 
infonnation comprises traffic loss ratio per Class of 
Service (CoS) per destination. 

12. The network element operable in an IP network as 

set forth in claim 5, wherein said QoS parametric 
infomiation comprises queuing delay infonnation. 
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13. The networi< element operable In an IP network as 
set forth in claim 1 2, wherein said queuing delay in- 
formation comprises average depth of said buffer 
queues on said ingress and egress cards. 

5 

14. The network element operable in an IP network as 
set forth in claim 5, wherein said QoS parametric 
infomiation comprises traffic flow jitter infonnation. 

10 15. The network element operable in an IP network as 

setforth in claim 14, wherein said wherein said QoS 
parametric infonnation comprises available band- 
width per link. 

'5 16. The network element operable in an IP network as 
setforth in claim 5, wherein said plurality of counters 
associated with said egress card include a counter 
for monitoring the number of packets transmitted 
per egress card per queue type for each neighbor- 
20 ing element associated with said networic element 
In said IP networic. 

17. The network element operable in an IP network as 
setforth in claim 5, wherein said plurality of counters 

25 associated with said egress card include a counter 
for monitoring the number of bytes transmitted per 
egress card per queue type for each neighboring 
element associated with said networic element in 
said IP network. 

30 

18. The network element operable in an IP network as 
setforth in claim 5, wherein said plurality of counters 
associated with said egress card include a counter 
for monitoring the number of packets dropped per 

35 egress card per queue type for each neighboring 
element associated with said networic element in 
said IP network. 

19. The networic element operable in an IP networic as 
40 setforth in claim 5, wherein said plurality of counters 

associated with said egress card include a counter 
for monitoring the number of bytes dropped per 
egress card per queue type for each neighboring 
element associated with said networic element in 
45 said IP network. 

20. The network element operable in an IP network as 
setforth in claim 5, wherein said plurality of counters 
associated with said egress card include a counter 

50 for monitoring average queue depth of said buffer 
queues per egress card. 

21. The network element operable in an IP network as 
set forth in claim 5, wherein said plurality of counters 

55 associated with said egress card include a counter 
for monitoring the number of times a particular buff- 
er queue crosses a packet discard threshold asso- 
ciated therewith. 
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22. A method for processing Quality of Service (QoS) 
parametric information in a networl< element oper- 
able in an Internet Protocol (IP) network, wherein 
said network element includes at least one temni- 
nating line card operable as an ingress card sup- 
porting an incoming communications link, at least 
one temilnating line card operable as an egress 
card supporting an outgoing communications link 
and a switch fabric disposed between said ingress 
and egress cards for supporting a plurality of virtual 
ingress/egress pipes (VIEPs) therebetween, com- 
prising the steps of: 

receiving incoming Infomriation on said incom- 
ing link of said network element; 
detemilning in an ingress portion a network 
processor system disposed on said ingress 
card whether said incoming infomnation per- 
tains to an IP-based service; 
responsive to said detennining step, propagat- 
ing said incoming infomnation to an egress por- 
tion of said network processor system via said 
switch fabric, said egress portion Including an 
embedded processor operable to perfonn a 
plurality of IP-based QoS (IPQoS) monitoring 
operations and for processing said incoming in- 
formation into processed information; and 
transmitting said processed Information to said 
egress card via a select VIEP for routing said 
processed information on said outgoing link to 
a neighbor in said network. 

23. The method for processing QoS parametric infor- 
mation in a network element operable in an IP net- 
work as set forth in claim 22, wherein said IPQoS 
monitoring operations include policing said incom- 
ing infonnation at said ingress card. 

24. The method for processing QoS parametric infor- 
mation in a network element operable in an IP net- 
work as set forth in claim 23, wherein said policing 
step comprises measuring said incoming infomna- 
tion against an expected behavior profile associat- 
ed therewith. 

25. The method for processing QoS parametric infor- 
mation in a network element operable in an IP net- 
work as set forth in claim 22, wherein said IPQoS 
monitoring operations include a plurality of flow con- 
trol steps for effectuating bandwidth management 
for said switch fabric. 

26. A Quality of Service (QoS) monitoring system for a 
network element operable in an Internet Protocol 
(IP) networi<, wherein said network element in- 
cludes at least one terminating line card operable 
as an Ingress card for supporting an incoming com- 
munications link, at least one temninating line card 



operable as an egress card for supporting an out- 
going communications link and a switch fabric dis- 
posed between said ingress and egress cards for 
supporting a plurality of virtual ingress/egress pipes 
5 (VIEPs) therebetween, comprising: 

a policing structure associated with said in- 
gress card for monitoring incoming traffic on 
said incoming communications link with re- 

10 spect to an expected traffic profile associated 

with said incoming traffic; 
a buffer acceptance and flow control module 
associated with each of said ingress and 
egress cards, said buffer acceptance and flow 

IS control modules operating to manage traffic 

flows associated with said VIEPs through said 
switch fabric, wherein said traffic flows are op- 
erable to be effectuated with resource reserva- 
tions allocated in said switch fabric; and 

20 a traffic shaping and scheduling module asso- 

ciated with said egress card for scheduling and 
shaping outgoing traffic on said outgoing com- 
munications link. 

25 27. The QoS monitoring system for a network element 
operable in an IP network as set forth in claim 26, 
wherein said policing structure, said buffer accept- 
ance and flow control modules, and said traffic 
shaping and scheduling module operate in concert 

30 with a plurality of counters associated with said in- 
gress and egress cards to provide QoS parametric 
infonnation necessary to effectuate a Service Level 
Agreement between a network service provider op- 
erating said network element and a subscriber for 

35 services available in said IP network. 

28. The QoS monitoring system for a network element 
operable in an IP network as set forth in claim 27, 
wherein each of said buffer acceptance and flow 
40 control modules is associated with a local conges- 
tion indicator for effectuating a packet discard 
mechanism to throttle buffers queues correspond- 
ing to said traffic flows in said switch fabric. 

45 29. The QoS monitoring system for a network element 
operable in an IP network as set forth in claim 28, 
wherein said buffer acceptance and flow control 
module associated with said ingress card is opera- 
ble to be controlled at least in part by a feedback 

50 control signal generated by said egress card's local 
congestion indicator. 

30. The QoS monitoring system for a network element 
operable in an IP network as set forth In claim 29, 
55 wherein said feedback control signal comprises a 
per-flow congestion threshold value. 
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